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IDENTIFIERS 
ABSTRACT 

The Classification of Secondary School Course^ (CSSC) 
is a coding activity to translate the ^diverse . course titles which 
appear on Jtrariscripts and master schedules of students into common 
terms*, permittiag meaningful comparisons to be made about these 
students and allowing secondary school coursework to be cpnsiglered as 
a coherent fa'ctor in statistical studies\ The CSSC facilitates this 
task byl aggregating under a unique main cfourse title all similar 
course t^les and assigning each main ti'tl-e a' unique six-digit code. 
A test of intercoder ^reliability was designed, conducted and analyzed 
to test the usefulftess of the CSSC as a coding tool and develop 
,recommendati.ons on coder training and the Coding task. This report 
summarizes the following activLties integr'ai to the test: (1) coder 
selection and train ing; 2 ) training methodology; 'and (3) coding task 
and analysis results . .Whi le there i s /a^ f^i r amount of variability in 
the coding,^ there are a" number of st jfcraight-f orward ways to reduce it 
and achieve nearly perfect codings. Some of these techn iques involve 
training enhanceiflertts^ie .g. , expanding the cross-reference list) and. 
sotne involve procedural approaches (e.<g. , using multiple coders). 
(Author/PN) ' " - • 
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INTRODUCTION 

r 



The Classif IcatPon of Secdndary School Courses (CSSC) was developed by Eval- 
uation Technologies Incorporated for the f^atlonal Center for Education Sta- 
tistics (NCES) under contract number 300'%81-0312. It will be used In the 
coding. of transcripts of a national sample of high school students as part 
of the High School and Beyond longitudinal, study. The goal of the NCES 
coding activity Is to translate the diverse course titles which appear on* 
transcripts and master schedules of students In the study Into^common terms 
permitting meaningful comparisons to be made / about these students and 
allowing secondary school coursework to be considered as a coherent factor 
In statl-st^l ca I studies. ^ 

The CSSC facilitates this task by aggregating under a unique main course 

title all similar course titles and assigning each main title a unique 

six-dlgit code. Coders will be able to Identify program areas and Indl- 

i " ' 

vldual' courses within them by matching course tittes from student data with 

tho codes and titled In the CSSC, The CSSC slx-dlglt code 1^ based on. the 

structure of the Classification of Instructional Programs (CIP) J T[\e first 

fqur digits correspond to the two-and four-digit .program areas In the CIP, 

fhe last two digits represent unique courses. 

To t^st the usefulness of the CSSC as a coding tool and develop rfecommenda- 
tlons on coder tral n'l ng , and the coding task, ETI designed, conducted, and 
analyzed test of Intercodfer reliability. This, report suriimarizes the 
following activities Integral to the test: 

• Coder setectlon and training 

• Training methodology . r 
m UDS^hig task and analysis results. . 



Mai Itz, Gera! d S. A Classification of I nstructjlonal Programs. U.S. 
Department of Edu^catlon, Office o# Education Resedrch and Improvement, 
National , Center for Education S-*-atlstIcs, NCES 81-323, Washington, D.C.: 
U.S. Government Prfntlng Off Joe, 1981. 
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I I . NETHODOLGY 



A. CODER SELECTION 

/ 

Three, coders and one alternate were Selected for the rel lablll-h/ study. 
The alternate-WHs~mcluded In cas^ one of the other coders failed to com- 
plete the ti^o-week study. All four: coders met the criteria of having an 
American high- school dlplana and ^minimum of two ful i years of college 
education. 'All coders were temporary employees: three were from Potomac 
Temporaries of Arlington, Virginia, and were public school teachers during 
the academic year; one was a temporary summer"^ empi oyee of Evaluation 
Technologies Incorporated who was a f ul Itlme undergraduate student during 
the remainder of the year, The^ following chart describes the four coders: 



■Coder 
~ 1 

2 

3 

4 



27 
25 

26-^ 



Education " 
B.A. 
B.A. 



2 yrs'Grf: coI4*g? 
32' M. Ed,.l.e<hreation 



Occupation 

High School History Teacher 

Elementary J School Teacher 

•Student - - ' 

High School S6clal Studies 
Teacher 



Sex 
F 
F 
M 
M 



Coder number 4 was randomly selected to be the' alternate. None of the 
coders were 1 nfprmed' that on ly three gets af data would be used unti I after 
al l of them had completed -the coding. Tho coders were p'al d by the hour. 

B. TRAIN i'nG - ' ' i> 

i. Tra In lng_ Materials _ . ^ • • 

The materials, used during the training session Included: ' , 



er|c 



• The Training Outline - See Appendix /\ 

• A Cla-sslf Icatlon of Secondary School Courses, Final Draft, 
June 18, 1982 

9 Training protocols with sampi e- course titles for practice cgdlng 
- See Appendix B " ' ' 

V 

i 

• Catalogs correspfondlng' to- the training protocols 

^ • .* 

\ ..} ' ■ . ^ 

• Reminder list of^speclal Instructions -See Appendix C 

o ' ' . ^ 

© Diagram "Making a^Codlng Decision" - See Appendl^ 

2. Training Procedures - - ' ^- * . 

The coders were trained on the 'first day In an elght--hour training session 
on how to assign slx-dlglt' codes to secondary course titles using tfae CSSC. 

The training session be^an with a 30-mInute I ntrcfductlon to the training 
session, the test, and the Intended use. of the CSSc! Tn' the NCES High SchtJol 
and-Beyorid longitudinal study. The next 30^f^rrfutes feopflnued with a page- 
by-page review of the' CSSC,' highl Ightlng each sect'lon as ^to Information 
Included, such as, the alphabetical Index anti the short Table of Cross 
i?eferences and the main bod'y of six-dlgit coded courses.^ After a 15^Inute 
break., the coders were -Introduced tq a sample protocol, for practice .In 
assigning the six-dlgit codes, the '50 secondary >f«chooI' catalogs to be used 
as the Wee of^the courses, and the coding procedure. The coding pt-oce- 
dure consisted of the coder entering his/her name, the staft time, a slx- 
dl'glt code drawn from the . CSSC,^a check mark noting - any reference to 
catalogs, and the finish time.. The coders were asked to maintain the ord.9r 
of the protocoI.s and ref ralTTf rom d'isCussIng coding decisions among them- 
selves, and to complete the. protocol^* on which they were worklo'g before 
taking a break o'r leaving- for the day. Coders were g I veri guidelines for> 
coding^ tracked and leveled courses.' After' the lunch break, the' coders 
spent the remaining fqur hour.s on'three vincreasing.ly difficult practice 
trials and reviewed their coding decisions and problems In a group with :the 
trainers. 



C. TEST OF THE CSSC ; c . . .■ 7 ^ • 

The coding of the titles Included In the test of the CSSC began on the. day 
following the training session. 

1. Material , -.- ^ , . ^ / 

The coders used the' final draft of the CSSC (A Classification of Secondary 
School Courses^ Final Draft, June 18, 1982) to code each of 1,000 titles. 
The 1,000 titles were chosen^from 50 catalogs of seconclary school courses 
supplied by NCES. Twenty courses^were chosen f rom each cata I og by choosing 
a randan starting pplnt and then picking titles evenly^ spaxed throughout 
the catalog. The twenty course titles from a catalog were listed on a 
protocol Identifying the catalog and leaving spaces for the coders to Indl- 
cate their coding decisions, whether they referrod to the course descrip- 
tlon In the catalog, and their start and stop tlme^^A sample protocol 'Is 
shewn as Exhibit 1. Thus each coder used the CSSC, 50 protocols containing 
20 titles ^ach, and the 50 catalogs. Coders wefe also permitted to use the 
.reminder list of special 'instructions (see Appendix C) and the diagram 
"Making a Coding Decision" (see Appendix D) that w^e distributed and 
discussed during the training session. An end-of-study questionnaire was 
completed by epch coder Immediately after completing the coding ^f the last 
protoco I ■. 



All coders cpded the 50 pro^tocols In the same order. The sessions took | 
place over an eight-day period, four days In, one week and four days In the 
following week. A three^day wetfekend Intervened.- In order" to reduce th?' 
need fpr -hfo ^ders to use a single catalog at tfie same time, the coders 
started, ^ne hour apart on 'the first-day. Each coder worked on one protocol 
at a-tlme, entering the six-digit, code for the title on the line'next te 
the course title. When' "I n doubt about a I I sted ^ourse title, /the coder 
referred to , the appropriate secondary school catalog and/ noted this 
reference with a check on the .'protocol ^sheet. The coder^s/ntered their 
start and stop times on the pratocol and did not take br^ks during the 
coding 6f a protocol. They .were encourage,^ to take shor/ breaks between 



Coder No*. 



Catalog No. 



School: J<lng $ton High ScliooK North Carolina 
Time started^ 



Course Name , ^ ' 

1. Old Testament 

2-. Accounting 1 . 

3. English 10, A^dvanced 

4. Literature Highlights 

5. Modern American Literature 
. 6. Crafts 

?• Music Theory 
♦ 

8. Spanish 2 

9. Calculus 
10. Archery 
11- Soccer 

12. Economy of Mo,dern America 

13. Salesmanship 

14. Te'chnical Drafting !• 

15. Auto Mechanics 2 

16. Business ^aw 
I/- ShortShand 1 

18- English 12, Advanced 

19- Modern English Literature < 

20- Vocabulary Development 



Code 

1. 

2. 

3. 

4. 

5. 

6. 

7. 

8. 

9. 
10. 
11. 
12. 
13. 
14. 
15. 
16. " 
17. 
18. 
19. 
20. 



Reference to 
cata'log 



Time completed: 



Exhibit f. 
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Typical Protocol 
-5- 

9 



protocols. Coders were not allowed tq discuss coding decisions ,untl I after 
the last coding session was complete. After completing each protocol, a 
coder handed It to the test monitor. % 

When a coder completed the flfti-eth protocol, he or she was given the end- 
of-test questionnaire to. compl ete.,., . When all the coders were finished, a 
prel^lmlnary summary of the data ^ dlsbussed\ with them and additional re- 
actions were solicited In an I nf ormal debr I ef 1 ng .session. 

Because the coders were working very Intensely and quickly, on the ,^^ond 
Godtng dap the test monitor cautioned the coders to walk around between 
protocols/ln order to hel p themsel ves maintain a high rate of efficiency. 
When totaNoutput fell to seven protocbls per day on. the third day, the 
mo.nltor told the coders that' th^ breaks were becoming too long. The number 



of protocols per day -Increaged 



although the average amount oj time between 



starting and completing a protdcol Idid not seem to be affected. 



Ml. RESULTS AND DISCUSSION 



A,. CODING TIME 



Coders wore serlou:. and attentive to the task and. adhered fo t^e 'constraint 
not^^to discuss the coding untIM after all coders had finished. Coders did. 
not' ask the monitor any questions aboui. selecting the codes during the 
coding sessions. Because' o^ the variations In the tl-tllng styles and the 
amount^ of descriptive material Included In the catalogs from different 
schools, It was not possible for -the three coders to' mal ntaifl" consi stency 
of pace from protocol to protocol. The average time per protocol 33 
minutes with a minimum of ten minutes and a maximum of 55, minutes. The 
time each coder used to code each protocol Is"shown In Exhibit, 2. ' By the 
end of the study,- the coders were comfortably coding r^I nd to ten protocol's 
M80 to 200 titles) per day each. 



B. ACCURACY 



Coder accuracy was determined by comparing each coder • s" response^ on a' 
sample of course titles to the correct classification for those titles as 
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' Coder 


^ . 4/ 




Protoco ! y 


1 


2 




3 


V 'Averdge 




t 

' 12- 


28 


27 ' 


25.7 


02 ' 


35 . 


' 49 


35 


- 39.7. 


03 


36 


53 


38 


• ^42.3 


04 


38 


48 


50 


\ 45.3 


05 


48 


71 




44 


54.3- 


06 


, .28' 


46 ^ 


29 


34.3 • 


■ 07 


35 


39 


61 


45.0 


■ 03 ■'• 


40 


40 


30 


,r 36.7 


09. ■ 


• 32 


35 


25 


30.7 ■ 


10 


37 


40 


55 


44'. 0-. . 


11 ' 


. 40 




25 


30.0 .' ■ 


12 


51 


\ 50 
^38 


30 


43. 7' 


13 


35 


35 


36.0 • . 


14 


20 


\16 


■ ' 15 


17.0 


• 15 


■ -34 . 


24 - 


30 


'29.3 


•1,6 


27 


30 ■ 


20 


" 25.7 


17 


41 


' -53 


40 


44.7 


18 


45 


32 


45 


47.3 


19 




■ 35 


32 


34.7 


20" 0 


49 


50, 


30 


. i 43-. 0 


21 


' 36 . 


21 




13 


■ . 24:7 


22 


26 


31 




.30 


29.7 


23 


. 39 


3^ 




^ 30 


. 34.7 • ' 


2'4 


29 


42 




25 


32.0 


25 


' 31 


. 3^ 




..' 25 ' 


\ .'30.3 


'26 • ■ 


37 




60 


.39.7 


27 


23 . 




13 . 


17.0. 


23 


31 


30, . 


25 . 


' 28.7 . 


29 


24 


' 25 


20 


23.0 


30 


48 


42 


3Q 


, MO-.O 


.31 . . 


21 ' 


lOi 


30 


20.3 


32' 


23 


35 ' 


40 


'32.7^ 


33 


32 


26 


35 


31.0 ' 


"34 ' . 


- >9 


- '32 


35 


■ 35.3 


35 ; 


50 


29. , 


20 ■ 


33.0 


, 36.. 


43 


27 i 


30" 


33.3 


- 37 

♦ - 


21 


20 ! 


20 


20.3 , 


•38 


25 


- .33 ; 


25 


' ■ 27.7"" " 


.39 


17 ■ 


22. • 


' 25 


21 .3 • 


40 


22 


23 • 


•20 


Z1.7 ' , 


41 . • 


. 27 • 


.40 


15 


27.3"* 


■ ■ • 42 , 


62 


30 


=55 


49 . 0 


43 . 


. .44 


15 ■ 


.20 


=•'26.3 


/ 44 


■ 17. 


25 


, 20 


20.7 


• 45 


20 


27 


15 ' 


20.7 




31 


25 


20 


.25.3 


47 ' ■ 


35 


52 


52 


46.3 


* 

A Q 




MO 




/in n 


. 49 


38. 


30 


25 . 


31.0 -. 




35 ■ 


25 


30 


30'. 0 


^ Average ' 


34 . 2 


" '55.8 


50.5 


, 32 . 8 • / 



* — — ; — ' ' — : — ' r-" ^ 



Exhibit- 2. Time to'Code Each Protocol by Cocior fn Minutes 



-determined by -fv/o CSSC experts."' Fifty titles. we chosen for ..the. accuracy 
check- by selecting .one titje' from.each of the fifty protocols'. Titles were 
selected that do not correspond , exact I y to the titles In the CSSC Index,. 
This represents' a- more d I f f I cu ft- set of coding decisions than those /found 
on the average protocol., because protocols frequently have many titles that 
exactly match those In the Index. Th^ "two expert coders classified the 50 
titles Independently an'd then met i-o resoj ve <the! r differences. 

The percent of agreement at the two-digit^ four-dlgHy and siK-dlgIt code 
level s, of each of the^i^hree coders _ w I th .>^he correct 'code Is shown' In 
E}<hlblt 3. ^ " 

Average agreement at the 4o~dIgIt level was ,^3 percent; at the four-digit 
level was 71 percent, .and at the six 'dTg It' I eve I was 43 percent.' These 
percentages can be 'taken as I ower bounds o'n 'the coders' accuracies since 
bourse- titles that exactly correspond to CSSC titles should rarely.be 
miscoded. If the coders had bee,n allowed to discuss their declsl.ons ^wfth 
each other, we would'also expect the amount of agreement to "be higher. The 
two expert coder^ had s I gn I f 'Leant amounts of divergence at the four- and 
stx-dlgit levels before conferring, ai||iough they had no difficulty In 
rpsolving their differences. ■ V ' „■ ' ' 

C. . I NTERCODER RELIABILITY • \ v 

\ 

• \._ 

Intercoder.'rel labll ItT^^as measured on ' the \f I rs+ ten protoco'ls (2CLD 
titles), the remaining protocols (800 titles),, Ud^overal I ddOO titles). 
Two sets of measures were constructed: • • 

* The percent. of 'agreement between each of the three possi b I e' pal rs' 
1 , * °^ cpders a^t the two-, four-, and six-dlgit code levels 

« The percent of codfes on which thr'ee,. tw5; or, no code'rs agreed at 
the twoV, four-, and SIx-dlgIt levels. : 

he^e me^asures a rd>, displayed In Exhibits 4 and's. By all measures th^ 
^greemenl- among the^^toders Improves between the beg 1 n^nl ng,.- of the study (the 
10 protocols) Vnd the end of the study (the remaining protocols). 





Leve 1 


Coder 


2-digit 


4-digit 


6-digit 


1 


90 




54 


2 


78 


Or 

..72,. 


42 

- Q 


3 

\- 


82 




32 


Average 


83 


71 


43 



/■ \ ■ 

/ )/ 

Exhibit 3. Percent of Two-, Four-, and Silx-DJ,^it Code Levels on 
V/hich each Coder Agreed with Expert Code Assignments 
on ^50 Hard Titles, 
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Coder Pa i r 


s . Level 


.2-dig.it 


4-digit 


6-^digit 


•Prg^focols 1-10 (200 Titles) • 




P.^ir 1-2 
Pair 1-3 / 
Pair 2-^ 


J- 

91.5 
90.0 
88.5 


86.5 ■ 

85.5 

82.5 


.72.5 , 

63.5 

.63.0, 


. . Protocols n-50 (800 Titles) 




Pair 1-2 
Pair li-3 
Pai r 2-3 


94.5 
94.0 
92.5 


92 . 5 
89.. 0 
86.8 • 


74.3 
71 .3 
65.8 


' f:-Overal 1 (1 ,000 Titles) ^ 


Pair 1-2 
Pair 1-3 
Pair 2-3 

1 


93.9 
93.2 
91 .7 


91 . 1 

88.3 . 
85.9 


73.9 

69.7 " ' 
' ' 65.2 



I 



Exhibit 4. Percent of Two-, Four-, and Six-Digit Code Levels on Which 
Each Pair of Coders Agree by Protocol Group and Overall. ■ 
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Code Level 


Number Agreeing 


3 


- 2 


0 • • 


Protocols 1-10 (200 Titles) 

«=t; — — . • 


2-digit 
4-cjigit . 
6-digit 


86.0 
79.0 
56.0' 


12.0 
17.5 
31 .0 


2.0 
3.5 
13.0 


* 

Protocols 1 1-50 (800 Tit les) _ - 


2~dia1t 
4-digit 
6-digit 


90.6 " 

84.6 

60.5 


9.1 . 
14.1 
29.8 


0.3 
1.3 

9.8 


■ Gv^ral 1 CI ,000 Titles) . 


2-digit ■ 

4-digit 

6-digit 


'89.7 
83.5 
59.6 


9". 7 
14.8 
30.0 


0.6 

1.7 ' 

10.4 



Exhibit 5, Percent of Two-, Four-, and Six-Digit Code Levels on'which 

Throe, Two, and No Coders Agree by Protocol Group and Overall* 



As expoctod, the overa! I • percentage of agreement among the pairs of coders 
Is considerably higher than each coder's percentages In the accuracy check. 
The overall palr-wfse average at the two-dfglt level Is 92.9 percent. At 
the four-dlgit level. It Is 88.4 percent. At the six-dlgit level. It Is 
69.6 percent. ^ ' 

• * 

As shewn In Exhibit 5, all three caders agreed at the two-digit Mevel 89.7 
percent of the time, and two of ,the coders agreed 9.7 percent of the jf^tme. 
•Either two or three coders agreed a't the two-digit level 99,4 percent of 
the time. At the four-digit >^vel, three coders agreed 83.5 percent of the 
time and two coders agreed 14.8 percent of the time. Either two. or three 
coders agreed at '^he four;:^dIglt level 98.3 percent of the time. At the 
six-dlgit level, three coders agreed 5^9.6 percent of the time and two 
agreed^ 30 percent of the time. Either two or three coders agreed 89.6 
percent of 'the time. Thus If the decisions of two out of three coders are 
accepted as satisfactory results of the coding task, agreement on a unique 
^code can be expected approximately 90 percent af the time at the finest 
level of cietall, 98 percent of the time at an Intermediate level of detail, 
and 99 percent of the time at the Instructional program area level (the 
two-digit code level). 

D. INSTRUCTH^QMA^ PROGRAM AREA SUMMARIES 

In order to determhie whether specific areas of the CSSC are causing *more 
probk'nis for the coders than others, , we tabulated for each Instructional 
program area (two-digit level \code) the number of times eaph-area was 
"selected as the classification for a title by all thr^ee-^eoders, / by two of 
the coders, and by only one of .the coders. ^>l-g'"Tnf ormatlon I^'tabled In 
Exhibit 6* The -table disti ngulshes^i>etw^en the two ways th^t.only one 
coder may select a code for a -tifte: ' • ^ 

■ -■■ / 

• The-ofher -hto coders agree with each other 
^ •. The other two coders disagree with each other. 
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*» 



1 



Ccle 



Nun&er of Coders Agrnelng 



Two 



Agree i ng 
Pa ir 



Other 
^Cod^r 



None 



Difficulty 
1 ndex* 



Number of 

Cata log 
References 



01 
22 
05 
C4 
C5 
•C6 
07 

ca^ 

09 
'lO 

u 

12 

V 

15 
16 
17 

la 

19 
20 
21 
22 
23 
24 
25 
26 
27 
28 
29 

^.r 

31 
32 
33 
34 
V:> 
36 
37 
33 
39 
40 
41 
42 
43 
44 
45 
46 
47 
48 
49 
50 



5 
4 
1 
1 

6 
18 
80 
9 
20 
6 
11 
1 • 
0 
. 0 
0 
75 
6 
0 
0 
60 
22 
, 3 

1 
1 

25 
92 
2 

2 
0 

17 
6 

16 
0 
2 
0 
9 
0 

35 
0 
3 
0 
0 

54 
5 

17 
47 
2 

122 



4 

1 : 

1 
1 
5 
1 

4, 

0 

1 

1 

1 

1 

0 
0 
0 
2 
0 
0 
0 
4 
1 
2 
14 
0 
0 
2 
2 

0 
2 
0 
5 
1 

5 
2 
8 
1 
1 

0 
3 
0 
0 
0 
0 
8 
2 
0 
3 
0 
8 



^1 

3 
0 
0 
5. 
1 
2 
2 
' 1 
2 
0 
0 
0 
0 
0 
0 
1 
0 
0 
3 
2 
0 
6 
0 
1 

1 . 

i 
1 

0 
8 
1 

13 
2 
' 7 
2 
4 
3 
1 

0 
5 
0 
1 

0 
1 
7 
0 
0 
5 
0 
4 



C 



0. 

0 

1 

l^ 

1 
1 

0 
0 
0 
0 
^ 0 
0 
0 
0 
0 
0 
0 
0 
0 

6 ^ 

0 ' 

0 
0 

2 
0 
0 
0 
4 

1 
1 

0 
2 
0 

1 

»o 

0 
0 
2 
0 
0 
0 
0 
2 
0 
0 
0 
0 
0 



0.2000 
0.7500 
1.0000 
0.0000 
0.8333 
O.lllf 
0.0375' 
0.2222 
Or05QO 
e.3333 
0.0000 
"^.OOQO 

o.ooob 

0.0000 
0.0000 
0.0000 
0.1^67 

■ 0.0000 
0.0000 
0.0500 

, 0.0909 
0.0000 
0.0541 
0.0000 
1 .0000 
0. 1200 
0.0109 

. 0.5000 

' 0.0000 
6.0000 
2.0000 
0.8235 
0.3333 
0.5625 
2.0000 
2.5000 
3.0000 

o.-iui 

0.0000 
0.2000 
0.0000 
0.3333 
0.0000 
1.0000 
0. 1667 
0.0000 
0.0000 
0.1004 
0.0000 
0.0328 



697 



97 



97 



18 



0. 126? 



^ I ndox 



Number in "Other Coder" Column and Number in "None" Column 



Number in "Three" Column 
If "Three" Column has a zero, 1 is used in the denominator. 



Exhibit 6. Number of "CI ass i f i cat Ions on which T-trree, Two A No Raters Agree 
and The Difficulty Index and th'6" Number of Catalog Refernces by 
Two-Digit Cod6 Level (Instructional Program Area) 
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As. a measure of the coding dlfflcull^ of .each Instructional program area, 
we computed a dlfflcuIfY Index for each Instructional area, by dlvl;dlng the 
, number .of times specific titles were assigned. to an Instructional area by a 
single coder by the number of times specific titles were 'assi gned to that 
area by 'all three coders. To ppevent division by zero, the divisor was 
arbitrarily set to one .If the. Instructional area was never chosen unanl- 
.mously. This dl f f Icu I ty I ndex ' Is Included In the sixth column of the 
tab|e. Exhibit 6 also displays the number of times the coders referred to 
the catalogs for titles I n each area, since the number of titles In each 
Instructional area varied widely, the Individual 'counts are of limited 
utlKlty.' The tota-l of 1191, however, represents almost 40 percent of the 
3000 \^dlng- decisions made by th0 three coders. Indicating that the coders 
made substantial use of the catalogs. 



there are eight Instructional areas with a dlfficulfy In4&x-of 1 or 
greater: " ' ' 





Area Code 


Area -Description - • 

/ 


/ 


« 

03 


Renewable Natural Resources 






Lfbrary and Archival Sciences 




30 


Multl/lnterdlsclpl I nary Studies 




31 


Parks and Recreation 




35 


I nterpersonal Ski 1 1 s 




36 


leisure and Recreational Activities 




37 


Personal Awareness f 




44 


Pubt Ic Affairs 



These accounted for seven perceiaf of ^ the titles coded. They appear to have 
been chosen for courses that simply did not fit well elsewhere. Their, high 
variability Is associated with their low frequency of occurrence. 

Nine Instructional areas had Indices of zero because/ they were never 
selected: 
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of these casq.s two or three of the coders chose*the same two-digit code and 
th^re was never a case of total disagreement: - • ' . 

^ Area Co(j,g Area Description ^ ' ^ 

♦ 

Architecture and EnvI>ronmental Desl gn 
Computer and Information Sciences' 
^2 Consumer, Percson^l, and MIscel I aneous Servl ces 

16 , . Foreign Languages! ^ . 

22 . Law \ * . 

24 I ' Liberal/General NStiidles 

46 Construction Trades 

47 Mechanics and Repairers* • • 
49 Trqtnsportatlon and Materia! Moving 

Of these nine areas, only areas 11, 16, and 47 were chosen more than 20^ 
times. 



U 
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^ • Area Code Area Description ' - ■ ' \ ' 

13 Education '* ^ 

14 Engineering 

15 ' . Engineering and Engineer I^ng-Rel ateS^chnolpgles 

18 Health Sciences ' ^ 

19 Home Economics ^ 
29 MM Itary Technologies. \ 

39 Vheology *^ . / ' • - • 

41 Science Technologies 

43 r Protective Services - \^ ■ ^ 

Most of these are clearly to6 advanced for most secondary aohool 

curr Icu la, * * 



There were another nine Instructional areas with' Indices 9^) zero, • In each 



There were seven areas w.lth indices between 'zero and O.'l indicates that ttie 
coders had very little disagreement. These areas are: 



Area Code 


Area Description 

■ ^ 




. Business and Office 


09 


Commun Icati ons 


20.^ 


Vocational Home Economics 


21 


Indus+rlai Arts 


23 


Letters 


27 


Mathematics 


50 


Visual and Performing Arts' 


•' 47 . 


Mechanics and Repairers 


49 


Transportation and Material Moving 



Each of 'these areas was chosen between 62 and 386. times, ' ' 

E. RESPONSE VARIATION ' 

* ■ 

There appeared to be at least seven different reasons for var.latlqn In 
y coder response. 

* , ■ 

\ ^- Level As?I^nment. Although coders agreed at the four-digit level,, they' 

\^varLed"Tn Interpretation of the level at the fifth and sixth digit level. 

\^For example. Protocol 2, title 17, "Economics." The codes assigned Were 

15.0602 Econctnics^" and Economic Problems; 45.0601 Economic Theory, Basic; 

\5.0611 Econom Ics, Col I ege. ', 

A\s[?milar situation occurred In Protocol 3, title 12, ' "Env Ironmental 
BlAlogy"- and Protocol 4, title 10, "Practical Math." /Th.Is source of 
yariyation could be ameliorated 'by developing gui delink sheets giving 
Information +hat distinguishes among, levels. Increased emphasis In the 
training should be placed on reviewing differences between levels. 

2. YgQue Secondary S chool Course Titles. Some -.condary school courses 
had vague titles; for example, "Literary Highlights." The course descrlp- 
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tlbn "included both British and Arherlcan literature. Since no code was 
, avail abl^f or an^al I-Inc| usive course, 'coders agrefed at the ' t.wo-dlgit 
"Utters" level, but'varled at the four- and' six-dlgit levels. A similar 
situaflon occurred at^Protocol 4, title 7, "Executive High School Intern." 

-Such courses will Invariably be Included In student data, The CSSC could 
not disaggregate every Imagrriabie course combj rfatlon.- In this situation 
the cod'e^s can only place cdurses tp the best of their Judgment, 
referring to the school catalogs and ^keywords In the CSSC. 

^' CgpgriC TItlg-s. in some Instances, the CSSC uses the generic title, 
for example, "Aquatlcs"\instead of a specific form of acquatlcs such as 
"Skin Diving." Protocol 38, title 16, listed the course title "Skin 
Diving." Coders placed It under 34.0161 "Physical Education Leadership 
training", 36.0161 "Aquatics" and 31.0200 "Outdoor Recreation, Other." ■ 

Coders need to' be trained to search for generic titles when the specific 
course title Is not* listed. A cross reference I 1st 'for such generic tjties 
could be added to the general list of cross references. 

j^PmbinQd, Concepts., Several courses combine concepts from two different 
programs. If Is difficult, to know whether to place the' course under the 
traditionaj heading (e.g.. History) or I n ^new' I y developed program areas 
such' as area studies or In mu I tl/ I nterdl scl p I I nary studies. This conflict 
is reflected In Protocol 13, title 13 "History and Ph 1 1 osophy of Sciences" 
which was- variously placed In 30.0411 Humanities, 45.0311 Archaeology, 
40.0100 Physical Sciences, Other. Similarly, Protocol 6, title 14 "General 
Science" was placed In 26.0611 Ecology, 30.0111 Science^ Unified, and 
26.0151 Fle.ld Biology. In the same manner Protocol 10, title 1 "American 
Studies" was placed In 45.0809 ' American History, Basic, 05.0103 American 
Studies, General, and 45.0822 American Inquiries.^ Finally in Protocol 10, 
tittle 4 "Technology "and Environment" was placed In 36.0621 Environmental 
Science, 03.0211 Conservation and Regulation, and 45.1131 Sociology, 
Issues. The Cross Reference List should be expanded to Include references 
to courses of' comb I ned concepts. 
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5-, -Olliiirgnces between Pronram^. in some cases, courts were placecT 
according to a course title that was listed but In a program that was not 
appropriate.. For example Protocol 3, title- 15 "Cooperative Office 
Management" was placed In 07.0742 Office Education 2, Cooperative; 07.0741 
Office Education 1, Cooperative; 32.0107 Cooperative Education 2. The 07 
Is the Business Program Category and the 32 is Basic Skills under "Personal 
and SccIa'I Development" Program' Category. In a^ similar way,' Protocol 5, 
title 5, "World CIvI I Izatlon" Is "l isted both under 45.08 History "and 30.04 
Humanities and Social Sciences by different coders. 

To llml,t this source 'of variation, the definition of the areas, ^"^p^rtlcu- 
Iarly*07 and 32 needs to be reemphaslzed for coders. 

6. Un^gar Course Oblectlves." Some catalogs list a course as "Drama" and 
do not clarify whether the course Is the'readlng of'^dramg, the writing of 
plays, ^or the actln^g of 'dramatic IKerq^ture. An example of this 
Js found In Protocol 6, title 8, "Beginning Drama". This'course was codQd 
both as 23.01, reading of drama and 50.05, acting of drama. Further^ 
ciarif Icatlorl and separation between program areas at the four-digit level 
should be made-. .That may be beyond the scope of the CSSC and more directly 
related to the problems of definition within disciplines as well as 
secondary school curriculum. 

^' Lqw Frequency Course Tltlfi.g, Occasionally a course will appear that Is 
unfamiliar to^ the coders and does n^t appear In the Index. 'This will be 
the case for highly Innovative and unusual secondary school courses that do 
•not fit traldltlonal categories well. For example, Protocof 6, title 17, 
"Peer Counseling" was coded both In 42.05 Counseling Psy.chology and 33.01 
CItlzenshlp/CIvIc Activities, This source of variation is Inherent In the 
variability of secondary school offerings and cannot readily be 
Gl Im Inc. ted. 

IV. CONaUSlONS AND RECOMI^ENDAT IONS 

The^baslc use of the CSSC can be easily taught t^nexperlenced. coders In a 
one-day. "training session. The coders can then proceed to code course 
titles at the rate of approximately 1.5 minutes per title. 
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J . While there Is a fair amount of variability In the coding there are a 
number of straight-forward ways to reduce It and achieve nearly perfect 
codings. Sane of these techniques Involve training enhancements and some 
Involve procedural app4;€)aches. 

■ A. TRAINING ENHANCEMENTS 

^ . Place Ad ditional Emphasis on the^ Content of the Instructional Are^^ s, 

The training as presented Is adequate for coders with the backgrounds of 
those used In this study. Because we hired them on a temporary basis dur- 
ing the summer, we were able to find school' teachers to do the coding. 
This may not be t^e case for a long term coding project that takes place 
J ' during the academic year. Since non-teachers will be^less familiar with 
content of the various Instructional areas, It^fould be wise to place addV 
tlonal emphasis In the training . on the content^xrf ;hhe^maJor Instructional 
areas. y • 

2- .Expand the Cross-Re ference L I .t. As noted In the discussion, a number 
of sources of variation could be counteracted by developing* an expaaded 
C^05s-Ref erence List to include references' to generic titles and courses 
with combined concepts. 

^" .Ingreqs e Instruction on Determlng Level s> Much of.^the variation netted 
.resulted frcm disagreements among the coders on 'the chofde^ of the fifth and 

sfxt4i: ill gltV for course with multiple, level • option^.' More detailed 
, guidelines should'be de\^e!oped for determining levels. Practice on using 
• these guidelines should be included In the training. - t 

B. PROCEDURAL APPROACHES, 

1 1. . Using Multiple gptjer'^. .Having two cr three coders code the same 

material will result In a high .degree of unique codings. Differences can 
then be resolved through discussions among the coders. Our experience 
^ suggests that trained coders will have little dlffrcuTt/ resolving their 
differences. Exhibit 4 Indicates that two coders will agree at the slx- 
dlglt level about 70 percent of the time, leaving about 30 percent of the 
-titles to be resolved through discussion. 
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Exhibit 5 shows that tv/o or three out of 'three coders will agree at the 
s^Ixrd%I^level approximately '90 percent of the time. Only th6 ten percent 
of the -i^^tles that all three coders disagreed on^would need to be 
dl scussod. " ' 

Presumably r^ular dl scusslons. among the coders would reduce futiire varia- 
tion In the coding decisions. At the very leasts multiple codings with 
discussion of ^\fferences should be used for the first few we§ks after the 
Initial tral nIngA When to stop multiple coding should be a function of the 
level of agreement Vnong the coders. 

Lot Code rs !Talk to Each Other and to an Expert. For .the Intercoder 
re! lability study we Iitslsted that al I coding decisions be Independent. 
This requirement would beVounter product I ve for a fuM-.scaie coding effort 
and should be dropped. Somdv permanent staff member should be designated to 
become the resident expert to whom^the temporary coders can turn to f6F\>' 
guidance In resolving dIfflcuitX cases. ' 




APPENDIX A 
TRAINING OUTLINE 
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aASSIFICATION OP .SECONDARY SCHOOL COURSEsV 
CODER TRAINING, RELIABILITY TEST, AND 'DATA ANALYSi^ PLANS 



I. TRAINING PLAN 
Training Program Goal 



As ,3' result of .an eight hour trai nl ng session, faur tral nees wh I be 
prepared "to assign six-dl.git codes to secondary course titles In order: to 
provide da.ta for' a rellablj Ity test of the CI assi f I cati on ' of Secondary 
SchoQl Coursas (CSSC). 



Training Objectives . * ^ 

As a result of trafnlng^ the trainees wlTl be able 'to: 



^ • Descrl'be the purpose of the^ coding task 

> ■ : 

•''Identify the dffferefit sect I on s| of the .CSSC 

• Identify the^vcrfous parts of- the course title unl^t for a course 
. Jn the CSSC*^ ^. 



• Follow proper coding procedure? In matching course titles listed 
on prepared protocol s wl th Related six-dlgtt codes frcm the 
'CSSC. : . ' - i ' . * ' ^ 



iRoqu ^r cd ^^aterlals 



• Training outline detalMng tral^n I ng. content and training approach^ 

• Six copies of thq comp feted CSSC 



• Six copies ot training protocols cental nl^ng. samp I e coyurse titles 



for practice coding 



« six copies of coder reminder list of special Instructions : 
, Six copies of the diagram "Making a Coding Decision." 

c . - 

TralQlno Agenda 

» * * ( 

A, Introduction to Training and Test . (30mln.) 

1. Review format for the training ^session, working hours, time 
i ^ sheets, physical arrangements, etc, for the period ot employment. 

•2. Summarize all participants^ backgrounds and roles to establish 
an environment of ^utual understanding, 

3. Describe the CSSC and Its Intended use In the NCES High 
School and Beyond longltud'inal study, 

4* Outline the coding task- and the purpose of the reliability 
'test, 

B, Present the CSSC (50 min.) 

1'. Highlight main points from the CSSC introduction, 

^2. Describe, using examples, the layout of the main body , of the 
CSSC, 

3, Explain the alphabetical Index, using examples, an-d demonstrate 
how it relates to the main body ^f the CSSC. 

* Explain the use of The List of Instructional Program Categories 
. V " ' and the table of Cross- References, 

.V' _. • : _ • . - 
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Break (15 min.) 

Descrlbe'the Coding Task (2 hours 15 mln.) 

1. Inspect a sample protocol Identifying: space for six-dlgit code; 
space for Indicating If reference was made to catal og; spaces to ' 
record start/f I nfslT'tlmes'. 

2. Explain use of I oca I'^cata I ogs^ with examples, describing 
relationship of catalogs to protocols and use .of catalogs as 
backup to coding decisions. » 

3. Outline coding approach, using examples and the diagram of a 
coding decision. Coders should be Instructed to: 

m Code every course title on protocol sheets with a full six- 
d I g I t code 

• Allow' adequate time for each Coding decision, as speed Is 
not to be emphasized over accuracy of coding 

•Use the "other" category only after making a thorough search 
of the taxonomy for coding alternative's 

•Code all bilingual, special education, or gifted course 
tltl'fes as If they were regular versions of the courses. 

4. " Describe the coding pirocedure. Coders should be Instructed to: 

i Record start/finish times of each protocol 
6 Maintain the order of the protocols 

• Ask questions of the test monitor at any time 

• Retrain f rem d IscussI ng coding decisions among themselves to 
avoid biasing the test results » 



w Complete the protocol tney are working on Defore taking breaks 
or leaving for the day. 



Provide coders with guidelines for coding "tracked" and "leveled" 
courses, , 

In cod I ng ."tracked" courses. I.e., courses I abel ed w Ith 
designators such as "Remedial," "Below Grade Level ," '"Standard, " 
"College," "Honors," etc*, coders should be Instructed to follow 
these gul del I nes: 

• If the course title to be coded Is designated as a track, apd 
the esse course title fs not divided Into tracks, do not use 
the track information In coding the course 

« If the course title'to be coded Is not , tracked, and the CSSC 
course Is divided Into tracks, code tnp course with the 
standard or middle track course code. 

In coding "leveled" courses. I.e. courses labeled wlrh designa- 
tors such as "First Year," "Advanced," or with numbers such as 
"Drafting 5," coders should be Instructed to follow these guide- 
i I nes: 

* 

• If the course title to be coded has no level designator, and 
the esse Includes only leveled courses, code the course with 
the ninth grade or level 1 course code * 

• If the course title to be coded hah a level designator, and 
the esse course^ is not leveled,! do not use the level 
designator- In coding the course ' 

• If The course title to be coded has a level designator higher 
than the l^vel used In the CSSC, code the title with the 
highest CSSC level , course code available. 
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6, Demonstrate the coding tasks using several e><anTp[es listed on*' a 
demonstration protocol. .... ^ 



E. Break (1 hour) 

F. Conduct Coding Practice Trials (4 hours) 

1. Four^ coders independently match- ten sample course titles 
presented on a trial protocol with six-dlgit codes from the CSSC 
anrf record their decisions. The ten coding decision are then 
discussed as a group. 

2. The coding practice task Is repeated with a second more difficult 
trail protocol of ten course titles. The coding results are 
discussed as a group. 



0/ 
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3. The need for more piisc^-Ice Is determined, and. If necessary, 
third" trial protocol Is employed for additional practice*- 



G. Concl ude Training 



1. Provide I nstruct^Ion^s.'--f5r beginning the reliability test on the 
f ol I ow I ng mpi^nTng. 
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Coder No. 



Schebl : Springfield Public Schools 



Catalog No. .,^^1- 



Time started: 



Course Name ^ , 

1. Sculpture 

2. Accounting 235 . • 
3*^ Home Economics Occupations 434 
4_ Industrial English 

5_ Language 
g_ Smart Spending 
7 Mixed Choir 334 
g Special Chemistry 
g Religions of the World 
Child Care 

11. 

12;. 

'13. 

14. 

15. 

15. - 

17. 

18. ' 

19. - 

20.. -. ' 



Code 



1. 



2. 

3. 

4. 

5. 

6. 

7. 

8. 

9.' 
10. 
11. 
12. 
13. 
14. 
15. 
16. 
17. 
18. 
19. 
20. 



Reference to 
catalog 



Time completed; 
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Coder No. 



School : 



North H.S., Omaha, Neb. 



Catalog No. Practice §1 
(cat Old #49) 

Course Name 

1. Beginning Journal ism 

.2. Survey of Black Literature 

3. Math Analysis 

4. American Government 1 
'5, Office Procedures 

6. Foods and Nutrition 

7. Agribusiness I 

8. Electronic Music 
■ 9. Gym Aide 

10. Drivers Education 
11. 



12. 
13. 
14. 
15. 
16. 
17. 
18. 
19. 
20. 



/ 



Time started: 



Reference to 
catalog 




Time completed: 
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Coder No. 



Catalog No. Practice n 



School: Springfield Public Schools, Illinois 



Time started: 



Course Name 

1. Drawing 

2. Creative Writing 

3. Drama/Play Production 

4. Perspective Rra^nce 

5. Interior Decorating 

6 . Sma 11. Eng i ne Repa i r 

7. College Algebra 

8. Beginning, Guitar 

9. Environmental Science I 
10.^ Consumer Education 

11. 

12. . . 

13. ' . 
14. 

15. - 

16. * 

17. 
18. 

19, 
20. 



Code 

1. _ 

2. _ 

3. _ 

4. _ 

5. _ 

6. _ 

7. '.^ 

8. _ 

9. _ 

10. _ 

11. _ 

12. _ 

13. _ 

14. _ 
.15. _ 

16. _ 

17. _ 

18. _ 

19. _ 
20. 



Reference to 
catalog 



V 



Time completed: 
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Coder No. 

Catalog No. Pi^^ctice §3 
(Cat New 112) 



School : Kennedy. SHS Bloomington" Minn. 



Course Name 

• 


Code 


1. 


Personal Analysis 


1. 


2. 


Aviation/Aerospace A 


2. 


3. 


Photo Offset 


3 


4. 


Norwegian, Conversational 1 




5\ 


Chemistry, Organic 


5. 


6. 


Minnesota Environment 


6. 


7. 


International Relations ^ 


7- 


8. 


Sales and Marketing, Basic 


8. 


9.' 


Audiovisual technology 


\9. 




Horse Care and Stable Operations 


10. 


11. 


- 


11. 






12. 


13. 




13. 


14. 




14. 


15. 


» . f 


15. 


16. 




16. 


17. 




17. 


18. 




18. 


19. . 


\ ■ . 


19. 


20. 




20. 



Time started: 



Reference to 
catalog 



..Time completed: 
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^ Coder Informa-fkon Sheet 



THINGS TO REMEMBER \ 
r 

1. Code the protocols in-order from one to fifty. Make sure not to skip 
any course tit Ics. . . ' 

^ 2. Work at a steady pace. Do not. feel you have to rush. Be accurate in 
your. coding decisions. 

3* Complete the protocol you are workjng on'^before taking a break or 
leaving for the day. 

4. Write all code numbers carefully and clearly on the- I ines provided. 

5. Do not discuss the coding decisions you have made with other coders. 



6. You can read course descriptions in high school catalogs for any titles 
you are not sure of. Remember to mark the protocol each time you refer 
to a catalog. 

• . 

7. Record the start/finish times far each protocol. 

8. Folfow the steps outlined in the diagram ^'Making a coding*decision^'" unti 1 
you g^t familiar v/ith the coding procedure. 

9. Use the "other" code ",00" only after carefully searching for a more 
speci f ic code. 

10. It may seem to you at times that more than one code fcould be given .to a 
single course title. Decide which icode is the "best" choice, usind the 
local catalog course descriptions, the location of^ the course y/ithiV the 
^school catalog, and fhe keywords and alternate titles* in the CSSC a5\ 
''guides.' 

n. Askfor general advice from the test monitor. However, he/she cannot give 
you specific recommendations for which codes to use. 
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NCES . I ntercoder Reliability Test 
Qoder Information Sheet #2 



> CODINS';'»TRACKED'^ AND "LEVELED- COURSES 

In coding "tracked" courses, i .e. courses labeled Wivth designators 

such as "Refnedial," "Below' Grade Level," "Standard ,jy"Col I ege/' "Honors," 

etc,, QOders should follow^these guidelines: 



If the course title to be coded is designated as a track, and the 
esse course title is not divided irtro tracks, do not use the track 
'Information in coding the course. 

If the course t-itle to be coded is not tracked, and the CSSe course 
is divided into tracks, code thp course with the standard or middle 
truck course code. ' ^ 

in coding "I eve I ed" ^courses, i.e., courses labeled with designatprs such 
as "First Year," "Advanced," or with numbers such as "Drafting 5," ccrders 
should follow these guidelines: 

• If the course tit'^- to be coded has no level designator, and the 
Cese includes only leveled courses, code the course with the ninth 
grade or level -1 course code^i _ - ■ 

• If the course title to be coded has a level designator, arfd the 
esse course is not leveled,, do not use the level designator in 
codi'ng the course. 

• If the course title to be coded has a level designator higher than 
the IfTveT used in the eSSe, code the title with the highest level 
course code available in the esSC. 



mKIMG A CODING DECISION 



• Wr4tQ ,jown start 'time at 
be'5innin'5 of 03ch protocol 



StAPT 



Check 
Title 


esse 

(ndox 






Is Cojrso Title Listed? 

o 




Yes 
T 


Find Course TItIo 
In esse Usinq 
6-DIgit Code 







Ho 



Do Descriptors and 
A! tomato TI M us 
Fit the Coarse Title? 



Yes 



Write 6-OiTit Code 
On Protocol 



To ^ioxt 
C'juf Tit!? to Ue 
Coded 



Do You Understand 
Course Title? 





Check; Program Category 

List On Page And Turn 

To Those Areas I n 
esse That May Contain 
A Rel ated Coltrse 



No 



Can You. Find 
Course At the 
Leve 


A Related 
J 6-Dlglt 
? 


1 Yes 


Write 6-Oigit Code 
On Protocol ^ 






Go To Next * ' 
Course Title ,\o bo . 
Coded ' 







No 



No 



Read Description In 

Local, Catalog 
(Marl< Protocol For Use 
Of Catalog) 



Read Descriptions Of x 
4-Oigit Areas And Compare' 
With Local Catalog 
^ Description 



Choose Most Simi lar 
4^Digit Area And Code 
wnh "00" Code At 5- And 
6-Digits For "Other" 



Go To Next . 
Course Title to be 
" ' Coded 



ERIC 



Write Jywn ccrplotion tiro at end of protocol. 

CK*>c»< .t3 r.iK.e 5ur<> vou hive mri<Gd locril catalc] use. 



r 



/... 



AePilslDIX E 

RESPOi^SES TO COD^ FEEDBACK QUESTIONNAIRE 
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4. What woul^ri you like to see added to the training? 
Training sufficient. 
Explanations and examples clear. 

A little more comparative work so, you wouldn't feel so unsure^a^out 
coding at very beginning. 

N.A. ' • 

y 

5. What would you like to see removed from the training? 
, Nothl ng. , ' ^ ' ^ 

6. What other changes would you suggest for the training? 

None. ' . 

Researching protocols In morning only. Variation of work during 
breaks. More training on cross-referencing and alternative titles wJth 
groups not as academically prepared. 



7. Were the Individual course descriptions In the CSSC sufficiently 
detailed? 



A! ways 



A Most of the time 

About haLf the time 



Sel dom 



ERLC 



Never 



8. What three two-digit Instructional program areas' do you feel are the 



<3 



hardest to use? (You may refer to your CSSC when answering this 
quost.Ion.) 

03, 09, 20, 22 (not enough detail ) , 23, 26, 38 (not enough detail),, 45, 
and 46-49 (unfamiliar category). 



9. What parts of these areas caused the most problems? 

Courses very similar. Vague descriptions of school courses. 
.Descriptions too -complete or Incomplete. Uncertainty In choosing an 
answer. 



40.0711, 09.0400, .09.0442 and 20.0122-25. 



10, Please provide us with any other suggested Improvements In the CSSC. 

Bilingual classes In specific disciplines listed. Special Education 
courses might be developed. Let people work at own pace, to assure 
accuracy. Fixed payment on completion of project. ' * . 



11. Please sum up In your own words your reactions to the CSSC and this 
experiment.- 

Coder 1 , CSSC seems to be a very useful, and more often than not, a 
precise coding tool. Aside from the areas previously mentioned, the 
secondary school courses from our protocols were readily classified. 

Consideration might be given to the fact that ^d I fferent coders have 
(wIM have) different paces and wM 1 be able to ^acc^omplTsh" the Jo^~lnore~ 



quickly , (slowly) than others. ^Perhaps a payment (lump sum) for the 

task compieteH -/nMlH hn A f4-I_r_lQn+ v /?^y nf g| * |nrat|nj rftgniirrpg, 



ERLC 



4j 



The coding task Is tedious and breaks are Important/ Perhaps morning 
work, or evening work (part time) would be one w^y of avoiding worker 
fatigue, and consequently, accuracy (sic). 

I made use of the school catalogs often. ^ 



£Qdgr 2. I can see a need for" a cl^slf I cation system such as this 
national ly. I hope this document mej^s the goals you expected. ; 



r - X^Qd^r „i5.^ Can" be tedious at times and Interesting at times also^ An 
experience definitely. 



Cod^'r 4, The esse. does accomjpllsh what It Is designed to do. That ik; 
to provide the user, with a manual which standardizes academiL and 
voeat lona I courses H n-a cdmprehensl ve way^. 
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